Super-convergence: very fast training of neural networks using large learning rates

Leslie N. Smith; Nicholay Topin

doi:10.1117/12.2520589

10 May 2019 Super-convergence: very fast training of neural networks using large learning rates

Leslie N. Smith, Nicholay Topin

Proceedings Volume 11006, Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications; 1100612 (2019) https://doi.org/10.1117/12.2520589
Event: SPIE Defense + Commercial Sensing, 2019, Baltimore, MD, United States

Abstract

In this paper, we describe a phenomenon, which we named “super-convergence”, where neural networks can be trained an order of magnitude faster than with standard training methods. The existence of super-convergence is relevant to understanding why deep networks generalize well. One of the key elements of super-convergence is training with one learning rate cycle and a large maximum learning rate. A insight that allows super-convergence training is that large learning rates regularize the training, hence requiring a reduction of all other forms of regularization in order to preserve an optimal regularization balance. We also derive a simplification of the Hessian Free optimization method to compute an estimate of the optimal learning rate. Experiments demonstrate super-convergence for Cifar-10/100, MNIST and Imagenet datasets, and resnet, wide-resnet, densenet, and inception architectures. In addition, we show that super-convergence provides a greater boost in performance relative to standard training when the amount of labeled training data is limited. The architectures and code to replicate the figures in this paper are available at github.com/lnsmith54/super-convergence.

Conference Presentation

Citation Download Citation

Leslie N. Smith and Nicholay Topin "Super-convergence: very fast training of neural networks using large learning rates", Proc. SPIE 11006, Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications, 1100612 (10 May 2019); https://doi.org/10.1117/12.2520589

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available